(a) Field of the Invention
The present invention relates to a data processing system for stereoscopic 3-dimensional video based on MPEG-4 and a method thereof. More particularly, the present invention relates to a data processing system for stereoscopic 3-dimensional video based on MPEG-4 and a method thereof, in which the system and method has the added ability of being able to synchronize stereoscopic 3-dimensional video data.
(b) Description of the Related Art
MPEG-4 is a standard that includes streaming of an elementary stream that is encoded according to object, synchronization of an elementary stream, and elementary stream management for managing the relation and position between each elementary stream and information for specific applications.
Elementary stream data are divided into access units (AUs), which are the smallest units that can have timing information, and AUs located at the same time position with respect to an elementary stream are decoded by one DTS (decoding time stamp) and CTS (composition time stamp). Time stamps such as the DTS and CTS are used to precisely display a decoding point of AUs in a decoding buffer of a receiving terminal (DTS), or a point for performing composition of CUs (composition units) in a composition memory.
To satisfy the standards of the MPEG-4 system, a new system decoder model (SDM) was defined, and a flexmux for supplementing a conventional transmitting medium function and a synchronization layer for adjusting time synchronization were newly introduced.
Stereoscopic 3-dimensional video are able to realize various display such as 2-dimensional video display, 3-dimensional field shuttering display, 3-dimensional frame shuttering display, stereoscopic video, and stereoscopic 3-dimensional polarized display, and video data and synchronization information are different depending on each display type.
The MPEG-4 system is designated for 2D (2-dimensional) multimedia data processing and display. Standardization has only taken place with respect to buffer management and a timing model for 2D video display.
However, with respect to stereoscopic 3-dimensional video that include left and right images, in order to maintain compatibility with the MPEG-4 system and realize effective synchronization between left and right images, there are needed buffer management and timing management methods that estimate DTS and CTS information with respect to AUs of the remaining channels that vary according to the different display types described above using DTS and CTS information with respect to an AU of one channel, thereby reducing a synchronization time and complication of a decoding model of the left and right images.
FIG. 1 is a drawing showing a decoder model of a conventional MPEG-4 system.
As shown in the drawing, the decoder model for the conventional MPEG-4 system includes a DAI (DMIF-application interface) 1, a decoding buffer 2, a decoder 3, a composition memory 4, and a compositor 5. The DMIF is divided into a control plane and a user plane, which are designated with respect to a delivery layer of the MPEG-4. The control plane of the DMIF manages sessions and acts to send and receive messages between peers, while the user plane mainly performs the sending and receiving of data.
The DMIF designates interfaces into two layers of DAI and DMIF network interface (DNI). DAI is an interface provided to DMIF application programmers, and DNI is used within the DMIF and performs work of mapping DMIF signaling (DS) messages, which is a protocol defined between peers, and transmitting the same.
The DAI 1 includes a demultiplexer, and provides access with respect to streaming data input to the decoding buffer 2. At this time, streaming data received through the DAI 1 are structured using SPS. Each SPS is structured in a chain of packets that include one elementary stream. Each packet includes elementary stream data divided into AUs, timing with respect to each AU, and other additional information. An SPS data payload is input to the decoding buffer 2. The additional information is removed at an input terminal of the decoding buffer 2.
AUs are the smallest units that can have timing information. Explanation information with respect to each AU is determined in an encoding step. The AUs are transmitted by the SPSs for input to the decoding buffer 2. The decoding buffer 2 is used to store AUs used in the decoder 3. A buffer amount required by the decoding buffer 2 is already known in a transmitting end, and information with respect to a buffer amount of the decoding buffer 2 is known in a receiving end. However, in the case where, depending on the application, there is already a predetermined value of the buffer amount of the decoding buffer 2, the predetermined buffer amount is used in the transmitting end.
The decoder 3 receives the AUs from the decoding buffer 2 precisely at a designated time, and performs decoding of the AUs. CUs resulting from the decoding operation are then output to the composition memory 4. As a result, the composition memory 4 stores the CUs. A size of the composition memory 4 does not need to be separately designated in the specifications of the MPEG-4 system.
The compositor 5 receives CUs from the composition memory 4, and performs composition or presentation in the case where the CUs are AV (audio/video) data, and performs a skip operation if the CUs are not AV data.
FIG. 2 shows a timing model for a conventional MPEG-4 system.
As shown in FIG. 2, in the timing model of the conventional MPEG-4 system, a standard clock or a time stream is used to synchronize AV data transmitted by elementary stream data.
The time samples DTS and CTS are used to precisely indicate points at which the AUs in the decoding buffer 2 are decoded, and points at which the CUs in the composition memory 4 undergo composition. Accordingly, the time stamps are related to the AUs and the CUs.
First, the DTS designates a decoding time with respect to each AU, and AUs corresponding to decoding times must exist in the decoding buffer 2. At this time, decoding with respect to the AUs is performed at the instant of the times designated by the DTS.
Next, the CTS designates a CTS with respect to each CU. The CUs corresponding to the CTSs must exist in the composition memory 4. Further, a present CU must be able to be used by the compositor 5 in a predetermined interval of time existing between a corresponding CTS and a subsequent CTS of a CU.
FIG. 3 is a drawing showing a buffer model of a conventional MPEG-4 system.
With reference to FIG. 3, the buffer model of a conventional MPEG-4 system shows a buffer operation of the decoding buffer 2. Buffer management is performed on the basis of DTS and CTS to prevent an overflow and underflow of the AUs and CUs.
AU buffering is managed by DTS, and decoding is performed by immediate conversion into CU at the time of DTS. At this time, in the case where it is known a predetermined time interval between consecutive AUs exists, the decoding time is temporarily known, and when this is not the case, the decoding time is known by the DTS of a packet header. Further, in the case where it is known a predetermined time interval between consecutive CUs exists, the CTS is temporarily known, and when this is not the case, the CTS is known by the CTS of a packet header.
Accordingly, the MPEG-4 system outputs elementary streams output from a plurality of encoders to each SPS via synchronization layers to adjust time synchronization. As a result, each SPS is structured in consecutive packets containing one elementary stream. These packets include elementary streams, which are divided into access units (AUs), which are the smallest units that can have timing information as described above, and also include timing and other additional information with respect to each AU.
An SPS data payload is input to the decoding buffer 2 of the system decoder model. At this time, additional information is removed in an input terminal of the decoding buffer 2. The decoding buffer 2 is used to store AUs required in the corresponding elementary stream decoder 3.
The decoder 3 receives the input of AUs from the decoding buffer 2 precisely at designated times to perform decoding. The generated CUs are then output to the composition memory 4. At this time, time stamps such as the DTS and CTS are used to indicate the times when AUs in the decoding buffer 2 are decoded and times when CUs in the composition memory 4 undergo composition.
However, since time models and system decoder models in the structure of MPEG-4 systems are limited to performing a standardization process on 2D multimedia data, there have not yet been disclosed buffer management methods and timing models for supporting stereoscopic 3-dimensional video processing and various display methods thereof.